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The  purpose  of  this  study  was  to  determine  if  analysis  of  covariance 
or  analysis  of  covariance  with  Porter's  true  covariate  substitution  ad- 
justment or  neither  is  appropriate  for  analyzing  pretest-posttest  educa- 
tional  experiments  with  less  than  perfectly  reliable  measures. 

Monte  Carlo  techniques  were  employed  to  generate  two  thousand  sam- 
ples for  each  of  forty-eight  sets  of  conditions.     These  conditions  in- 
cluded six  combinations  of  reliability,  two  levels  of  sample  size,  two 
levels  pf  gain,  and  the  equality  or  inequality  of  pretest  means.     Each 
sample  was  analyzed  by  both  of  the  tested  methods.     The  proportion  of  re- 
jections for  each  method  of  analysis  and  the  number  of  times  the  £ 
statistic     for  standard  analysis  of  covariance  exceeded  that  for  analysis 
of  covariance  with  Porter's  adjustment  were  recorded. 
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The  hypothesis,  "The  sampling  distributions  of  the  test  statistics 
from  analysis  of  covariance  with  and  without  Porter's  adjustments  are 
the  same,"  was  rejected  at  the  .01  level  of  significance  using  the  sign 
test  for  each  of  the  forty-eight  sets  of  conditions. 

To  gain  further  insight  into  how  the  two  methods  of  analysis  dif- 
fered, four  factorial  experiments  were  conducted  using  either  the  com- 
puter generated  alphas  or  powers  as  the  criterion  variables.  The  first 
factorial  experiment  examined  analysis  of  covariance  when  the  mean  gain 
was  zero  in  both  groups,  thus,  the  computer  generated  alphas  were  used 
as  criterion  variables.  The  second  experiment  examined  analysis  of  co- 
variance  with  Porter's  adjustment  when  the  mean  gain  was  zero  in  each 
group.  Again,  computer  generated  alphas  were  used  as  criterion  variables, 
The  third  and  fourth  factorial  experiments  examined  standard  analysis  of 
covariance  and  analysis  of  covariance  with  Porter's  adjustment  when  the 
mean  gain  in  only  one  group  was  positive.  The  four  factorial  experiments 
included  three  factors.  These  were  reliability  at  six  levels,  sample 
size  at  two  levels,  and  the  equality  of  pretest  means  at  two  levels 
(pretest  means  equal  and  pretest  means  not  equal).  In  each  experiment, 
it  was  found  that  unequal  pretest  means  combined  with  low  reliability 
were  significant  sources  of  variation. 

Additional  study  showed  that  unequal  pretest  (covariate)  means  com- 
bined with  low  reliability  produced  misleading  results.  More  specifi- 
cally, when  either  method  of  analysis  produced  a  significant  £  statis- 
tic and  there  was  a  mean  gain  of  zero  in  both  groups  but  the  pretest 
mean  of  one  group  was  less  than  that  of  the  other,  the  adjusted  post- 
test  means  indicated  that  the  group  with  the  higher  pretest  mean  had 
the  larger  gain.  Likewise,  when  there' was  a  positive  mean  gain  in  the 
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group  with  the  lower  pretest  mean,  the  adjusted  posttest  means  indicated 
the  group  with  a  mean  gain  of  zero  had  the  larger  gain. 

The  results  of  this  study  combined  with  the  results  of  others 
concerned  with  analysis  of  covariance  where  the  covariates  are  fallible 
point  to  the  inadequacy  of  the  technique  as  it  now  exists.  These 
results  should  be  kept  in  mind  if  the  technique  is  to  be  used. 


CHAPTER  I 
INTRODUCTION 

.  Regardless  of  the  theory  or  hypothesis  being  investigated,  educa- 
tional researchers  must,  at  some  point,  relate  it  to  learning.  One 
common  element  of  most  definitions  of  learning  is  that  a  change  must 
take  place  in  the  learner  (DeCecco,  1958,  p.  243;  Hilgard,  1956,  p.  3; 
Hill,  1971,  p.  1-2).  Thus  most  investigations  in  education  are  concerned 
with  identifying,  measuring,  and  comparing  these  changes. 

In  the  past  decade,  analysis  of  covariance  has  become  a  standard 
procedure  for  comparing  groups  with  different  levels  of  ability.  The 
basic  derivation  of  this  procedure  requires  the  assumption  that  the  co- 
variable  does  not  contain  errors  of  measurement  (Cochran,  1957).  In 
recent  years,  several  researchers  have  recognized  that  this  assumption 
is  violated  when. the  qovari'ate  is  a  mental  test  score  (Lord,  1960; 
Porter,  1967;  Campbell  and  Erlebacher,  1970).  Lord  (I960)  and  Porter 
(1967)  have  proposed  adjustments  to  the  analysis  of  covariance  procedure 
to  use  when  the  covariable  is  fallible,  that  is,  when  measurement  error 
is  present  in  the  covariable.  The  focus  of  this  study  is  to  determine 
if  analysis  of  covariance  and  analysis  of  covariance  with  Porter's 
adjustment  are  different  for  a  fallible  covariate  and  if  either  is 
appropriate  under  varying  conditions  of  reliability,  sample  size,  gain, 
and  pretest  mean. 

The  Problem  of  Comparing  Groups  of  Differing  Abilities 

The  problem  of  comparing  groups  based  on  change  manifests  itself 
prominently  when  the  variables  are  mental  measurements.  Most  physical 


science  measurements  are  made  with  the  aid  of  some  type  of  physical 
instrument.  If  an  engineer  is  interested  in  the  weight  gain  of  a  metal 
before  and  after  a  galvanizing  process,  instruments  are  available  to 
measure  the  weight  of  the  metal  to  within  at  least  one  microgram.  On 
the  other  hand,  if  an  educator  is  interested  in  the  change  in  a  student's 
I.Q.  before  and  after  taking  part  in  an  experimental  program,  an  error 
of  5  points  or  more  would  not  be  uncommon.  Based  on  the  Wechsler 
Intelligence  Scale  for  Children  with  a  standard  error  of  measurement  of 
5.0  (Cronbach,  1970,  p.  222),  an  error  of  5.0  I.Q.  points  or  more  would 
occur  in  approximately  32  percent  of  the  measurements,  assuming  a  normal 
distribution.  If  the  actual  change  in  I.Q.  were  near  zero  for  an 
individual,  it  is  srery   likely  that  the  error  of  measurement  would  exceed 
the  change  itself. 

The  problem  is  compounded  further  as  the  groups  being  compared  often 
are  not  of  the  same  ability  level ,  Many  of  the  recent  programs  featuring 
innovative  teaching  practices  for  compensatory  education  are  available 
only  for  the  most  need^^  and  the  comparison  group  is  then  sampled  from 
the  general  population  of  students  (Campbell  and  Erlebacher,  1970). 
Programs  such  as  Head  Start  and  Follow  Through  have  probably  been  the 
victim  of  tragically  misleading  analyses,  such  as  in  the  Westinghouse/ 
Ohio  University  study  (Campbell  and  Erlebacher,  1970).  Personal  contact 
with  the  evaluation  of  Follow  Through  projects  has  emphasized  the 
magnitude  of  the  problem.  Warnings  about  the  inappropriate  uses  of 
existing  modes  of  analysis  have  come  from  several  sources  (Lord,  1960, 
1967,  1969;  Campbell  and  Erlebacher,  1970). 

Remedies  have  been  offered  (Lord,  1960;  Porter,  1967),  but  at  this 
time  there  is  no  conclusive  evidence  to  indicate  these  remedies  are 


appropriate.  Studies  concerning  the  robustness  of  the  analyses  to 
violated  assumptions  have  also  been  in  conflict  (Peckham,  1970). 

Methods  of  Comparing  Groups  of  Differing  Abilities 

Jvio   general  approaches  are  available  to  compare  groups  with  dif- 
fering abilities.  The  first  is  to  compare  the  average  change  in  one 
group  with  the  average  change  in  the  other  group  by  means  of  a  _t  test  or 
analysis  of  variance.  The  second  is  to  use  analysis  of  covariance,  where 
the  pretest  score  is  the  covariate. 

There  are  at  least  three  methods  of  computing  change  scores  to  be 
considered  if  the  first  approach  is  used.  One  method  is  to  obtain  raw 
difference  scores  by  simply  subtracting  each  pretest  score  from  its 
corresponding  posttest  score.  Another  method  is  Lord's  true  gain  (Lord, 
1956)  which  was  further  developed  by  McNemar  (1958).  Basically,  this 
method  requires  the  use  of  a  regression  equation  to  estimate  the  "true" 
gain  or  change  from  pretest  to  posttest.  A  third  method  of  measuring 
gain,  the  method  of  residual  gain ,  was  proposed  by  Manning  and  DuBois 
(1962).  This  method  involves  regressing  the  posttest  scores  on  the 
pretest  scores  and  obtaining  a  predicted  posttest  score  for  each  subject. 
The  indicator  of  change  is  the  observed  posttest  score  minus  the  pre- 
dicted posttest  score.  This  method  is  mathematically  equivalent  to  the 
standard  analysis  of  covariance  (Neel ,  1970,  p.  30-31),  the  second 
approach. 

The  second  general  approach  is  the  standard  analysis  of  covariance 
procedure  found  in  many  statistical  texts  (e.g..  Kirk,  1968;  Winer,  1971). 
One  of  the  reported  functions  of  a  covariate  is  to  adjust  the  treatment 
means  of  the  dependent  variable  for  differences  in  the  values  of  corres- 
ponding independent  variables  (Hicks,  1965).  Logically,  it  seems  that  by 


using  the  pretest  scores  of  groups  differing  in  ability  as  a  covariate, 
the  treatment  means  can  be  statistically  adjusted  for  the  differences  in 
pretest  means,  A  deficiency  of  this  method  is  the  fallibility  of  the 
covariate.  A  fallible  variable  is  one  which  is  not  measured  without 
error  or  with  perfect  reliability  (Lord,  1960).  Lord  (1960),  recognizing 
this  deficiency  in  the  use  of  analysis  of  covariance,  derived  an  adjust- 
ment to  compensate  for  the  fact  that  the  pretest  scores  v^ere  fallible. 
Porter  (1967)  modified  Lord's  procedure  for  more  than  two  groups  and 
empirically  investigated  the  sampling  distribution  of  Lord's  statistic. 
Porter's  solution  has  been  suggested  as  an  alternative  to  the  standard  *-•' 
analysis  of  covariance  when  the  covariate  is  fallible. 

Limitations  Used  in  this  Study 

This  study  was  designed  to  examine  the  characteristics  of  the  two 
covariance  methods  within  the  framework  of  simulated  situations  which 
may  occur  in  the  analysis  of  a  compensatory  education  project.  Realis- 
tically, several  limitations  were  imposed.  Even  with  these  limitations, 
the  two  methods  of  analysis  were  investigated  under  forty-eight  combi- 
nations of  differing  reliability,  sample  size,  mean  gain,  and  pretest 
means. 

The  study  was  limited  to  two  groups,  henceforth  called  the  gain  and 
no-gain  groups.  Investigating  more  than  two  groups  would  not  only  in- 
crease the  number  of  possibilities  appreciably  but  would  not  be  in  line 
with  the  objective  stated  previously,  that  is,  comparing  a  compensatory 
education  group  with  a  "comparison"  group.  The  levels  of  the  reliabil- 
ities being  investigated  are  also  limited  to  the  range  .50  to  .90.  Sel- 
dom do  mental  measurement  instruments  have  reliabilities  above  .90  and 
instruments  with  reliabilities  below  .50  are  generally  considered  in- 
adequate. 


The  levels  of  sample  size  were  10  and  100  subjects  per  group. 
These  quantities  are  representative  of  small  and  large  sample  sizes  en- 
countered in  educational  experiments. 

Furthermore,  the  true  score  and  true  gain  variances  will  be  fixed 
a  priori  with  the  true  gain  variance  always  four  percent  of  the  true 
score  variance.  These  figures  are  empirically  based  on  Follow  Through 
achievement  data  in  keeping  with  the  aforementioned  objective. 

Procedures 

The  study  compared  the  two  methods  of  analysis  under  forty-eight 
sets  of  conditions.  The  two  methods  were  analysis  of  covariance  and 
analysis  of  covariance  with  Porter's  adjustment  for  a  fallible  covariate. 

Three  levels  of  reliability  and  two  levels  of  sample  size  were  used. 
The  three  levels  of  reliability  will  include  situations  where  the  re- 
liability of  the  pretests  are  assumed  equal  and  situations  where  they 
differ  to  more  closely  simulate  compensatory  education  project  evalua- 
tion. Each  of  these  combinations  of  conditions  was  repeated  for  four 
separate  cases.  Case  I  is  where  there  is  no-gain  in  either  group  and  '^ 
both  groups  have  equal  pretest  means.  Case  II  is  where  there  is  gain  in 
only  one  group  (the  gain  group)  and  both  groups  have  equal  pretest  means. 
Cases  III  and  IV  are  with  and  without  gain  where  the  pretest  means  are 
different. 

For  each  of  the  forty-eight  sets  of  conditions,  two  thousand  samples 
were  computer  generated  and  analyzed  using  both  methods  of  analysis  and 
a  .05  level  of  significance.  Two  thousand  samples  provide  empirical 
estimates  of  the  fraction  of  type  I  and  type  II  errors  to  within  .01  of 
their  true  values  with  ninety-five  percent  confidence.  The  samples  were 


generated  from  an  assumed  normal  population.  The  generation  technique 
is  one  proposed  by  Box  and  Muller  (1958)  and  modified  by  Marsaglia  and 
Bray  (1964).  This  procedure  generates  random  normal  deviates  with  a 
mean  and  variance  of  0  and  1  respectively.  These  normal  variables  are 
then  transformed  to  attain  the  specified  means  and  variances.  The  IBM 
370/165  computer  of  the  North  Florida  Regional  Data  Center  was  used  for 
the  generation  and  analyses. 

Using  these  analyses,  two  research  questions  were  examined: 

1.  Is  there  any  difference  between  the  sampling  distributions 
of  the  test  statistics  of  standard  analysis  of  covariance 
and  analysis  of  covariance  with  Porter's  adjustment? 

2.  What  factors  affect  each  type  of  analysis? 

The  first  research  question  can  be  restated  in  terms  of  a  null 
hypothesis  for  each  set  of  conditions: 

The  sampling  distributions  of  the  test  statistics  from 
analysis  of  covariance  with  and  without  Porter's  adjustment 
are  the  same  for  each  set  of  conditions. 
This  hypothesis  was  tested  for  each  reliability,  sample  size,  gain,  and 
pretest  mean  combination. 

The  second  research  question  was  examined  in  four  factorial  experi- 
ments. Each  factorial  experiment  included  the  factors  of  reliability  at 
six  levels,  sample  size  at  two  levels,  and  equality  of  pretest  means  at 
two  levels^  The  first  experiment  consisted  of  examining  standard  analy- 
sis of  covariance  with  computer  generated  alpha  values  as  the  criterion 
variables.  The  second  examined  analysis  of  covariance  with  Porter's  cor- 
rection also  using  computer  generated  alpha  values  as  the  criterion  var- 
iables. The  third  and  fourth  factorial  experiments  examined  analysis  of 


covariance  with  and  without  Porter's  adjustment  respectively,  using 
computer  generated  powers  as  criterion  variables. 

Relevance  of  the  Study 

This  study  is  designed  to  compare  two  recommended  methods  of  ana- 
lyzing educational  experiments  under  known  simulated  conditions  which 
are  presumably  realistic.  The  results  should  indicate  whether  either 
of  the  methods  is  appropriate  for  its  recommended  use,  thus  giving  edu- 
cational researchers  some  direction  when  faced  with  the  choice.  If  both 
methods  were  shown  to  be  appropriate,  the  study  could  determine  which 
one  is  superior.  The  study  should  also  indicate  if  either  method  of 
analysis  is  appropriate  for  a  set  of  restricted  conditions,  e.g.  for 
only  certain  levels  of  reliability. 

Organization  of  the  Dissertation 

A  statement  of  the  problem,  a  description  of  possible  solutions, 
and  an  overview  of  the  procedure  of  this  study  has  been  included  in 
Chapter  I.  A  comprehensive  review  of  the  related  literature  is  provided 
in  Chapter  II.  This  review  includes  a  historical  overview  of  the  pro- 
blem and  a  detailed  description  of  the  methods  being  compared.  A 
description  of  the  procedures  followed  for  this  study  is  contained  in 
Chapter  III.  The  data  and  analyses  are  presented  in  Chapter  IV  and  a 
discussion  of  the  results  is  provided  in  Chapter  V.  A  summary  of  the 
study  is  provided  in  Chapter  VI. 


CHAPTER  II 
REVIEW  OF  THE  RELATED  LITERATURE 

Comparing  groups  of  differing  abilities  has  been  a  problem  for  some 
time.  The  literature  has  included  discussions  of  possible  solutions  for 
at  least  the  past  three  decades.  Many  of  these  discussions  have  been  in 
conflict  with  one  another,  and  there  is  still  no  general  agreement.  The 
evaluation  of  federal  compensatory  education  projects  in  the  last  decade 
has  intensified  the  debate  and  the  need  for  a  theoretically  sound  and 
practically  useful  solution.  Some  of  the  solutions  proposed  over  the 
years  are  matching  subjects,  gain  scores,  and  analysis  of  covariance. 
A  discussion  of  these  solutions  and  why  they  may  not  be  considered 
satisfactory  is  provided  in  this  chapter.  The  major  properties  of 
analysis  of  covariance  are  also  considered.  Because  Porter's  adjustment 
for  a  fallible  covariable  in  analysis  of  covariance  is  not  readily  avail- 
able in  the  literature,  its  derivation  is  provided  in  this  chapter. 

Historical  Review  of  the  Problem 

A  paper  by  Thorndike  (1942)  examined  in  detail  the  fallacies  of 
comparing  groups  of  differing  abilities  by  matching  subjects.  The  crux 
of  his  argument  was  that  the  regression  effects  were  systematically  dif- 
ferent "whenever  matched  groups  are  drawn  from  populations  which  differ 
with  regard  to  the  characteristics  being  studied,"  (p.  85). 

During  the  late  1950 's  and  early  1960's  the  literature  was  inundated 
with  papers  on  how  to  or  how  not  to  measure  gain  or  change.  One  of  the 
leaders  during  this  period  was  Lord  (1956,  1958,  1959,  1963).  His 


proposal  for  estimating  the  "true"  gain  was  originally  set  forth  in  1956 
with  further  developments  in  1958  and  1959.  McNemar  (1958)  extended  his 
results  for  the  case  of  unequal  variances  among  the  groups.  Garside 
(1956)  also  proposed  a  method  for  estimating  gain  scores  and  Manning  and 
DuBois  (1962)  presented  their  derivation  of  residual  .gain  scores. 

The  debate  over  what  techniques,  if  any,  should  be  used  for  measuring 
gain  or  change  was  at  its  peak  in  the  early  1970's.  Cronbach  and  Furby 
(1970)  concluded  that  one  should  generally  rephrase  his  questions  about 
gain  in  other  ways.  Marks  and  Martin  (1973)  underscored  the  Cronbach 
and  Furby  (1970)  conclusion.  O'Connor  (197.2)  reviewed  developments  of 
gain  scores  in  terms  of  classical  test  theory.  Neel  (1970)  employed 
Monte  Carlo  techniques  to  compare  four  identified  methods  for  measuring 
gain.  The  compared  methods  were  raw  difference.  Lord's  true  gain, 
residual  gain,  and  analysis  of  covariance.  Under  equivalent  conditions, 
he  found  that  Lord's  true  gain  tended  to  produce  a  greater  significance 
level  than  the  user  would  intend,  that  is,  a  higher  fraction  of  type  I 
errors  than  alpha. 

The  analysis  of  covariance  method  has  been  widely  recommended  by  a 
number  of  authorities  in  the  field  (Thorndike,  1942;  Campbell  and  Stanley, 
1963;  O'Connor,  1972).  Campbell  and  Erlebacher  (1970)  point  out  that  the 
Westinghouse/Ohio  University  Study  was  evaluated,  possibly  incorrectly, 
using  the  analysis  of  covariance.  Lord  (1960,  1967,  1969)  has  sounded 
a  warning  about  its  use  that  has  been  echoed  by  others  (Werts  and  Linn, 
1970;  Campbell  and  Erlebacher,  1970;  Winer,  1971).  The  warning  stressed 
that  the  analysis  of  covariance  requires  the  assumption  that  the  covariate ''''^ 
is  measured  without  error.  Lord  (1960)  has  proposed  an  adjustment  which 


10 


was  later  generalized  by  Porter  (1967).  The  adjustment  is  based  on  the 
substitution  of  the  true  score  estimate  for  the  observed  value  of  the 
covariate. 

Analysis  of  Co variance 

The  analysis  of  covariance  procedure  was  originally  introduced  by 
Sir  Ronald  A.  Fisher  (1932,  1935).  According  to  Fisher  (1946,  p.  281), 
the  analysis  of  covariance  "combines  the  advantages  and  reconciles  the 
requirements  of  two  widely  applicable  procedures  known  as  regression  and  ^ 
analysis  of  variance."  The  procedure  is  well  documented  in  contemporary 
texts  (Snedecor  and  Cochran,  1967,  p.  419-446;  Kirk,  1968,  p.  455-489;  : 
Winer,  1971,  p.  752-812).  Analysis  of  covariance  is  a  popular  technique 
in  both  the  physical  and  social  sciences. 

Among  the  principal  uses  of  analysis  of  covariance  pointed  out  by 
Cochran  (1957),  p.  264)  is  "to  remove  the  effects  of  disturbing  variables 
in  observational  studies."  It  was  thought  that  by  using  the  pretest  score 
as  the  covariate  and  comparing  two  groups  with  analysis  of  covariance, 
the  effects  of  different  pretest  scores  could  be  eliminated.  Lord  (1960) 
pointed  out  that  this  was  not  necessarily  the  case  when  the  covariate 
was  fallible.  The  assumptions  necessary  for  the  analysis  of  covariance 
are  the  same  as  those  for  analysis  of  variance  with  the  addition  of  the 
following: 

/ 

1.  The  covariates  are  measured  without  error.  / 

2.  The  regression  coefficient  is  constant  across  all  treatment 
groups  (Peckham,  1970). 

Violation  of  the  first  assumption  induces  a  bias  in  the  analysis 
of  covariance  because  of  "the  presence  of  'error'  and  'uniqueness'  in 
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the  covari ate,  i .e.  variance  not  shared,  by  the  dependent  variable.     If 
the  proportion  of  such  variance  can  be  correctly  estimated,  it  can  be 
corrected  for,"   (Campbell   and  Erlebacher,  1970,  p.   199).     This  position 
was  upheld  by  Glass  et  al .    (1972).     The  basic  algebraic  derivation  of 
this  correction  was  presented  by  Lord  (1960)  and  expanded  by  Porter  (1967] 

Analysis  of  Covariance  with  Porter's  Adjustment  for  a  Fallible  Covariate 

Lord's  (1960)  derivation  of  the  analysis  of  covariance  adjustment 
was  limited  to  two  treatment  groups  and  is  slightly  more  difficult  than 
is  Porter's   (1967)  procedure.     Thus,  Porter's  procedure  will  be  derived 
here  and  used  in  the  analysis.     Porter's  adjustment  is  based  on  the  sub- 
stitution of  the  estimated  true  value  for  the  covariate. 

Let  X  denote  a  fallible  variable  (e.g.   a  pretest  score) ,  r  an  esti- 
mate of  the  reliability  of  X,  and  T-j  the  true  value  of  the  variable  X^- . 
Let  T-j  denote  the  estimated  true  score  of  X-j .     The  definition  of  T-j  for 
each  individual   is 

(1)  Ti  =  X  +  r(Xi-X), 
or  T^.  =  rX^-  +  X(l-r), 

Porter  (1967)  derived  the  mean  and  variance  of  T-j  as  follows.  By 
definition,  the  mean  of  all  T-j's  is: 

(2)  T=  ^Ti^ 

IT' 

=  E[rX^+)r(l-r)] 
ii 

by  substitution.  Thus 

T  =  rX  +  NX-NrX  , 
N 

=  X , 
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where  N  denotes  the  sample  size  and  it  is  understood  that  i  is  summed 
over  the  values  1,  2,  ---,  N. 

Also,  by  definition,  the  variance  of  t  is 

(3)  S^  =  E(fi-X)^  ^ 

N-1 

=  r2s2 
X' 

and 

(4)  Sj^  =   z(fi-)()(Yi-Y)  , 

N-1 

where  Y  is  the  dependent  variable. 

Based  on  these  results.  Porter  (1967)  derived  an  analysis  of  covar- 
iance  procedure  replacing  the  fallible  covariate,  Xj ,  with  the  estimated 
true  score,  T^-.  The  analysis  of  covariance  requires  the  computation  of 
analysis  of  variance  sums  of  squares  for  the  dependent  variable,  the  co- 
variable,  and  on  the  cross-products  of  the  dependent  variable  and  the  co- 
variable.  Porter  (1967)  showed  that  the  use  of  estimated  true  scores  for 
the  covariable  did  not  affect  the  analysis  of  variance  of  the  dependent 
variable,  Y.  The  changes  found  by  Porter  (1967)  in  the  other  two  cases 
are  as  follows: 
For  the  analysis  of  variance  of  the  estimated  true  scores,  f, 

(5)  SS^^  =  lE(fij-f.j)2 

=  r2j:z(Xij-X.j)^, 
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where  SSj,,  denotes  the  within  groups  sum  of  squares. 


(6) 


SSn.  =  nE(t.-i-f..)' 


=  nE(X..-X..)^ 
where  SSg^  denotes  the  between  groups  sum  of  squares, 


(7) 


SSy.  =  ZE(Ti-j-T..)' 
T 


=  r2zzX?.+(l-r)E 


n      N 


where  SSj^  denotes  the  total   sum  of  squares,  n  denotes  the  number  of 
(T,Y)  pairs  per  treatment  group,  and  N  denotes  the  total   number  of  (t,Y) 
pairs.     In  a  similar  manner,  the  cross-products  sums  of  squares  are: 


(8) 
(9) 

(10) 


SSwfv  =  ^^^(Xij-X.j)    (Yij-Y.j), 


SSBfY  ^  "^(X.j-X..)(Y.j-Y..)^ 


SSt.     =  rEZX,-iY,--i+(l-r)E 


(i:Xij)(zY,-j) 


— J^ 


Thus,  the  adjusted  sums  of  squares  are 

(rSS   )2 
SSu'  =  SSi^      ^XY   , 
Y   ~2^7 " 


(11) 


SS,, 


X 

^XY 
SS,, 


(rSSw  +  SSn  )2 
(12)       SS't  =  SSt  -     XY     XY 


^^^\  \  SSb^ 


(13) 


SS '   =  SS'   -  SS ' 
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Note  that  the  adjusted  v/ithin  groups  sum  of  squares  remains  un- 
changed by  the  substitution  of  T  for  X  but  the  substitution  does  alter 
the  adjusted  total   sum  of  squai'es  and  consequently,  the  adjusted  between 
groups  sum  of  squares. 

A  question  arises  concerning  what  value  to  use  as  the  estimate  of 

reliability  in  the  formulae.     A  solution  to  this  problem  was  proposed 

by  Campbell   and  Erlebacher  (1970). 

In  a  pretest-posttest  situation  one  may  find  it  reasonable 
to  make  two  assumptions  that  would  generate  appropriate  common- 
factor  coefficients.     First,  if  one  has  only  the  pretest-posttest 
correlations,  one  may  assume  that  the  correlation  in  the  experi- 
mental  group  was  unaffected  by  the  treatment.     (We  need  a  survey 
of  experience  in  true  experiments  to  check  on  this.)     Second,  one 
may  assume  that  the  common-factor  coefficient  is  the  same  for 
both  pretest  and  posttest.     Under  these  assumptions,  the  pretest- 
posttest  correlation  coefficient  itself  becomes  the  relevant 
common-factor  coefficient  for  the  pretest  or  covariate,  the 
"reliability"  to  be  used  in  Lord's  and  Porter's  formulas,    (p.   200) 

This  recommendation  will   be  followed  in  this  study,  thus  the  corre- 
lation between  the  pretest  scores  and  posttest  scores  will   be  used  as 
the  reliability  estimate  in, the  formulae. 

S  umma  ry 

As  noted,  analysis  of  covariance  and  analysis  of  covariance  with 
Porter's  adjustment  have  been  recommended  by  several  authors.  Campbell 
and  Erlebacher  (1970)  computer  generated  data  for  two  overlapping 
groups  with  no  true  treatment  effect  and  concluded  that  the  analysis 
of  covariance  method  was  inappropriate  and  that  analysis  of  covariance 
with  Porter's  adjustment  should  only  be  undertaken  with  great  tenta- 
tiveness.  Porter  (1967)  computer  generated  data  to  compare  the  _F 
sampling  distribution  with  the  theoretical  £  distribution  using  his 
adjustment.  He  concluded  that  samples  of  20  or  larger  were  needed  to 
have  a  useful  approximation  to  the  theoretical  F  distribution.  He  also 
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found  that  the  estimation  degenerated  further  when  the  reliability  was 
less  than  .7.  No  study  was  found  which  compared  both  techniques  under 
similar  conditions  for  both  gain  and  no-gain  groups. 


CHAPTER  III 
PROCEDURES 


The  analysis  of  covariance  and  the  analysis  of  covariance  with 
Porter's  adjustment  were  compared  under  forty-eight  sets  of  conditions 
on  the  basis  of  computer  generated  data.  Six  combinations  of  reliability, 
two  sample  sizes,  two  levels  of  gain,  and  two  different  sets  of  pretest 
means  were  used.  Both  equal  and  unequal  reliabilities  were  used  in 
the  comparisons.  A  random  sample  of  two  thousand  observations  was  gen- 
erated under  each  combination  of  conditions  and  analyzed  by  analysis  of 
covariance  with  and  without  Porter's  adjustment. 

The  sampling  distributions  of  the  two  analyses  were  statistically 
compared  with  a  sign  test  for  each  of  the  forth-eight  sets  of  conditions. 
The  computer  generated  alpha  values  and  powers  were  then  used  as  the 
criterion  variables  in  four  factorial  experiments.  Subsequent  a  poste- 
riori analyses  were  performed  where  warranted. 

The  Model 


A  standard  model  was  used  to  represent  the  pretest  and  posttest 
scores  of  one  subject.  The  model  follows  the  traditional  measurement 
approach  as  found  in  Gulliksen  (1950)  or  Lord  and  Novick  (1968)  and 
extended  to  gain  score  theory. by  O'Connor  (1972). 

(14)  X  =  T  +  E^ 
and 

(15)  Y  =  T  +  G  +  E2 
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where      . 

X  =  observed  pretest  score, 
Y  =  observed  posttest  score, 
T  =  true  pretest  score 
G  =  true  gain, 

£■]=  random  measurement  error  in  pretest  score, 
£2=  random  measurement  error  in  posttest  score. 
The  following  properties  about  E-j  and  E2  are  assumed  to  exist: 
The  errors  E^  and  Eo 
i)  have  zero  means  in  the  group  tested, 
ii)  have  the  same  variances  for  both  groups, 
iii)  are  independent  of  each  other  and  of  the  true  parts  of 
each  test. 
It  is  further  assumed  that  T  and  G  are  independent  (across  subjects)  and 
that  all  components  follow  a  normal  probability  distribution.  These 
assumptions  parallel  Lord's'  (1956)  stated  and  implied  assumptions. 

Selecting  the  Reliabilities 

Reliability  is  related  by  definition  to  the  variances  of  the  ob- 
served scores,  the  true  scores,  and  error.  This  section  shows  how  that 
relationship  can  be  used  to  establish  desired  reliabilities.  The  basic 
definition  of  reliability  (Helmstadter,  1964,  p.  62)  is  given  by 
equation  (16)  where  the  symbol,  p^^,  denotes  reliability. 

(16)     p^x  =  4~4 

2   ■ 
°X   . 

Then, 


;i7) 


a2  =  o2(l_p   ) 


■1 


XX 


Choosing  the  variance  of  X  a  priori  to  be  100,  the  variance  of  E-\   can  be 
found  in  the  following  manner  as  the  reliability  of  X  assumes  different 
values: 


(18) 


-1 


100(1 -p  ) 
XX 


The  independence  of  T  and  E-]  in  (14)  implies 
(19)       "2 


^x  =  ^T '  i. 


Combining  (18)  and  (19)  and  solving  for  a^  yields 


(20) 


1  =  ^°Sx- 


The  posttest  variances  can  be  selected  in  a  similar  manner.  Based 
on  the  assumptions. 


(21) 


a^  +  a 


?  +  a2 


The  variance  of  the  gain  scores  is  chosen  in  the  manner  prescribed  in 
Chapter  I.  Then,  combining  (16)  and  (21), 


;22l 


4. 


+     2 


'T 


P|YY 


where  pyy  denotes  the  established  reliability  of  the  posttest  scores. 
Thus  it  can  be  seen  that  the  effect  of  selecting  specified  reliabilities 
can  be  obtained  by  selecting  the  variances  of  E-] ,  E2,  and  T  in  accor- 


dance with  (18),  (20) 


,  and  (22)  respectively. 


The  Regression  of  Y  on  X 

Recall  that  one  of  the  assumptions  necessary  for  analysis  of  co- 
variance  (Cochran,  1957)  is  that  the  regression  slopes  of  the  dependent 
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variable  on  the  covariate  must  be  equal  for  each  treatment  group.  Let 

gy-x  denote  the  regression  slope  for  one  treatment  group. 

Then, 

(Ferguson,  1971,  p.  113), 


(23) 

Py-X  =  PXY  •  ^  ' 
^X 

implies 

(24). 

ev.Y  =       "XY 

2^2 

"T  ""  °E^ 

The  covarianee  between  X  and  Y  is  equal  to  the  variance  of  T  since  it 
has  been  assumed  that  all  the  components  of  the  pretest  and  posttest  are 
independent  except  T rwith  itself.  Therefore 

2 

(25)  By-x  =    ^T    , 

2   2 

and 

(26)  gy.x  =  Pxx 

by  equation  (16).  Thus,  the  slope,  Y  on  X,  of  any  group  is  equal  to  the 

reliability  of  its  pretest.  If  the  reliabilities  of  the  pretests  were 

the  same  for  both  groups,  the  assumptions  concerning  slopes  would  be 
satisfied. 

From  equation  (16),  it  can  be  seen  that  the  reliability  of  a  test 
is  dependent  to  a  certain  extent  upon  the  variability  of  the  sample  for 

which  it  is  given.  In  equation  (16),  o^   is  in  both  the  numerator  and 

A 

denominator  of  the  fraction  with  a^   being  subtracted  from  the  numerator. 

2 

As  0^  increases  both  the  numerator  and  denominator  increase  by  the  same 

2 

amount  assuming  o^  is  not  changed  thus,  pxx  increases.  Two  groups  of 

equal  ability  would  likely  produce  similar  variances,  hence  similar 
reliabilities.  However,  the  technique  is  often  used  for  comparing 
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compensatory  programs  with  a  comparison  group  sampled  from  the  general 
population  of  untreated  children  in  the  same  community  such  as  with  the 
Westinghouse/Ohio  University  study  (Campbell   and  Erlebacher,  1970).     These 
comparison  children  tend  to  be  higher  in  ability  than  the  treatment  group. 
The  resulting  differences  in  variation  tend  to  produce  different  reli- 
abilities.    In  order  to  simulate  such  a  situation  this  study  used  dif- 
ferent reliabilities  for  the  gain  and  the  no-gain  groups  in  addition  to 
the  comparisons  when  the  reliabilities  are  equal.     The  reliabilities  for 
the  gain  group  are  decreased  by  fifteen  percent  which  roughly  approxi- 
mates the  reduced  variation  empirically  observed  in  Project  Follow 
Through  achievement  data. 

Selecting  Heans 

The  true  score  mean  was  set  a  priori   at  100  when  both  the  gain  and 
no-gain  groups  have  equal  means.     The  situation  in  which  the  gain  group 
has  a  lov;er  mean  was  also  analyzed.     In  this  case,  the  true  score  mean 
of  the  gain  group  was  set  at  80.     These  values  have  been  empirically 
chosen  based  on  Project  Follow  Through  data.     Based  on  the  assumptions, 

(27)  E(X)  =  E(T+Et)  =  y^, 

hence  the  mean  of  the  observed  pretest  scores  is  equal  to  the  mean  of 
the  true  scores.  Also 

(28)  E(Y)   =  E(T+G+E2)   =  y-^+yg. 

Thus,  the  mean  of  the  observed  posttest  scores  is  equal   to  the  sum  of 
the  means  of  the  true  scores  and  the  gain  scores. 

Clearly  in  the  case  of  the  no-gain  group  and  in  both  groups  where 
no  gain  was  used,  the  mean  gain  was  zero.     The  selected  value  of  yQ  for 
the  gain  situation  was  based  on  power  considerations,  that  is,  y^  was 
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chosen  such  that  the  power  of  an  £  test  for  analysis  of  covariance  was 
.50. 

A  linear  model  representation  of  the  analysis  of  covariance  for  two 
groups  is 

(29)  Y  =  eg  +BiX+e2W+e 

where  X  is  the  pretest  score  (covari ate),  Y  is  the  posttest  score,  and 
W  is  a  dummy  variable  designating  group  membership  (W=l  if  gain  group, 
0  if  no-gain  group).  Testing  the  hypothesis  that  62  ""s  equal  to  0  in 
equation  (29)  is  equivalent  to  the  F  test  for  treatments  in  the  analysis 
of  covariance  procedure.  It  can  be  shown  that  B2  i"  equation  (29)  is 
equivalent  to  the  mean  gain,  pg.  The  mean  of  the  posttest  for  the  gain 
group  is 

(30)  E(Yg)  =  30+^1^X3+^2 

where  Yq  is  a  posttest  score  for  the  gain  group  and  p^  "is  the  mean  pre- 

G 

test  score  for  the  gain  group.  Likewise,  the  mean  of  the  posttest  for 
the  no- gain  group  is 

(31)  E(Yng)  =  Bo+6iyXf^e 

where  Y|^q  is  a  pretest  score  for  the  no-gain  group,  pv       is  the  mean  pre- 

NG 

test  score  for  the  no  gain  group.     But  it  has  been  assumed  that  p^    and 
PXfjQ  are  equal.     Furthermore,  the  gain  group  has  mean  gain,  pg  and  the 
no-gain  group  has  mean  gain  zero,  thus 

(32)  pg  =  E(Yg)-E(YNg)  =  B2  ' 

Hence,  choosing  the  value  of  62  'that  yields  a  power  of  .50  is  equivalent 
to  choosing  a  value  of  pg  to  produce  a  power  of  .50  in  the  analysis  of, 
covariance  procedure. 
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This  power  can  be  obtained  from  the  following  probability  state- 
ment: 

(33)      Pr[t*>tJ  =  .50 

where  t_*  is  a  noncentral  t_  statistic.  This  expression  can  be  approxi' 
mated  by  the  substitution  of  a  z,  statistic  for  t*. 


(34) 


Pr 


^2-H  >  t 


=   .50 


Thus  a  value  of  60  can  be  chosen  such  that  the  substituted  z_ 
statistic  is  equal  to  t  Q25  with  the  appropriate  degrees  of  freedom. 
This  value  of  &2   ""s  the  value  of  the  average  gain,  \ir,   such  that  the 
power  of  analysis  of  covariance  is  .50  under  the  condition  of  perfect 
reliability.  In  order  to  find  this  value  of  yg,  a  numerical  expression 
for  or  is  needed. 

The  variance  of  63 >  of  »  can  be  approximated  in  the  following 

manner.  Karmel  and  Polasek  (1970,  p.  245)  state  that  a?  is 

62 


(35) 


2    2 

J-  =  ar; 
32    Y 


e(x-x) 


v^2 


[e(X-X)2][e(W-W)^]-[i:(X-X)(W-W)]^ 


Dividing  both  the  numerator  and  denominator  by  N^  and  substituti 


ng 


population  variances  for  sample  variances,  a?  is  approximately  equal 

62   ,     .      . 

to  the  following: 


(36) 


32, 


2       2 

^Y      °X 

N   2  2/   .2 
Ov  o,,  -{a^J 


'X   "W 


XW' 


The  quantity,  oy^^,   the  covariance  between  X  and  W  has  been  assumed  equal 
to  zero  thus,  in  equation  (36),  the  Ow's  divide  out.  Hence 
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(37)       a?  =  a? 


2- 
Under  the  conditions  assumed  for  the  model,  the  variance  of  Y,  '^Y, 

2 
is  equal  to  4.  The  variance  of  W,  °W  can  be  computed  to  be  .25.  Thus, 
2  2 

by  substitution,  o-  equals  .80  when  N  is  equal  to  20  and  ^g  equals  .08 

1^2  2 

when  N  is  equal  to  200. 

Hence,  for  N  =  20,  Mq  can  be  found  by  the  following  expression: 

(38)  yg  =  t  ^^^  ,^a;-     =  2.1]    /rm  =  1.88. 

20   ~-U^o,l/  ^2 

Likewise,  for  N  =  200,  yg  can  be  found  by  solving  equation  (39). 

(39)  yg    =t  025  igyaa  =  1. 97/708  =  .56. 

200    -u^o^iy/  ^2 

The  approximated  values  of  yg  were  tested  using  Monte  Carlo  generated 
variables  and  found  to  indeed  produce  a  power  of  .50. 

The  approximated  values  of  yg  under  each  set  of  conditions  are 
shov/n  in  Tables  1  and  2. 
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TABLE  1 

CONDITIONS  UNDER  WHICH  COMPARISONS  WERE  MADE  WHEN 
THE  RELIABILITIES  FOR'  BOTH  GROUPS  WERE  EQUAL 


GROUP 

GAIN 

GROUP 

PRETEST 

PRETEST 

SAMPLE 

MEAN 

MEAN  FOR 

MEAN  FOR 

RELIABILITY 

SIZE 

GAIN 

GAIN  GROUP 

NO- GAIN  GROUP 

.90 

10 

0 

100 

100 

.90 

10 

0 

80 

100 

.90 

100 

0 

100 

100 

.90 

100 

0 

80 

100 

.70 

.  10 

0 

100 

100 

.70 

10 

0 

80 

100 

.70 

100 

0 

100 

100 

.70 

100 

0 

80 

100 

.50 

10 

0 

100 

100 

.50 

10 

0 

80 

100 

.50 

100 

0 

100 

100 

.50 

100 

0 

80 

100 

.90 

10 

1.88 

100 

100 

.90 

10 

88 

80 

100 

.90 

100 

56 

100 

100 

.90 

100. 

56 

80 

100 

.70 

10 

88 

100 

TOO 

.70 

10 

88 

80 

100 

.70 

100 

56 

100 

100 

.70 

100 

56 

80 

100 

.50 

10 

88 

100 

100 

.50 

10 

88 

80 

100 

.50 

100 

56 

100 

100 

.50 

100 

56 

80 

100 
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TABLE  2 

CONDITIONS  UNDER  WHICH  COMPARISONS  WERE  MADE  WHEN 
THE  RELIABILITIES  FOR  BOTH  GROUPS  WERE  UNEQUAL 


RELIABILITY 

RELIABILITY 

GROUP 

GAIN 

GROUP 

PRETEST 

PRETEST 

FOR  GAIN 

FOR  NO- GAIN 

SAMPLE 

MEAN 

MEAN  FOR 

MEAN  FOR 

GROUP 

GROUP 

SIZE 

GAIN 

GAIN  GROUP 

NO-GAIN  GROUP 

.76 

.90 

10 

0 

100 

100 

.76 

.90 

10 

0 

80 

100 

.76 

.90 

100 

0 

100 

100 

.76 

.90 

TOO 

0 

80 

100 

.60 

.70 

10 

0 

100 

100 

.60 

.70 

10 

0 

80 

100 

.60 

.70 

100 

0 

100 

100 

.60 

.70 

100 

0 

80 

100 

.42 

.50 

10 

0 

100 

100 

.42 

.50 

10 

0 

80 

100 

.42. 

.50 

100 

0 

100 

100 

.42 

.50 

100 

0 

80 

100 

.76 

.90 

10 

1.88 

100 

100   '- 

.76 

.90 

10 

88 

80 

100 

.76 

.90 

100 

56 

TOO 

100 

.76 

.90 

■  100 

56 

80 

100 

.60 

.70 

10 

88 

100 

TOO 

.60 

.70 

10 

88 

80 

TOO 

.60 

.70 

100 

56 

100 

100 

.60  ~ 

.70 

100 

56 

80 

100 

.42 

.50 

10 

88 

100 

100 

.42 

.50 

10 

88 

80 

100 

.42 

.50 

100 

56 

TOO 

100 

.42 

.50 

100 

56 

80 

100 
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Generation  of  Random  Normal  Deviates  with  Specified  Means  and  Variances 
The  study  required  the  use  of  computer  generated  normally  distrib- 
uted random  variables  with  specified  means  and  variances.  Two  thousand 
sets  of  variables  were  generated  for  each  of  the  forty-eight  sets  of 
conditions.  Muller  (1959)  identified  and  compared  six  methods  of  gen- 
erating normal  deviates  on  the  computer.  A  method  described  by  Box  and 
Muller  (1958)  was  judged  most  attractive  from  a  mathematical  standpoint. 
According  to  Muller  (1959,  p.  379),  "Mathematically  this  approach  has  the 
attractive  advantage  that  the  transformation  for  going  from  uniform  deviates 
to  normal  deviates  is  exact."  This  method  was  endorsed  by  Marsaglia  and 
Bray  (1964).  They  modified  the  algorithm  to  reduce  central  processing 
computer  time  without  altering  its  accuracy. 

The  method  first  requires  the  generation  of  two  independent  uniform 
random  variables,  U^  and  U2,  over  the  interval  (-1,  1).  The  variables 


and 


Z^  =  U^[-2  ln(U^+U^)  /  {U^  +  ul)2^^^ 


I2  =   U2[-2  In(U^ui)  /  (U?  +  \ilW^^ 


will  be  two  independent  random  variables  from  the  same  normal  distrib- 
ution with  mean  zero  and  unit  variance.  The  variables  were  then  trans- 
formed to  have  the  desired  means  and  variances. 

Analysis  for  Comparing  the  Selected  Methods 

Each  of  the  two  thousand  sets  of  generated  data  for  the  forty-eight 
sets  of  conditions  was  analyzed  by  analysis  of  covariance  and  analysis 
of  covariance  with  Porter's  adjustment.  The  number  of  times  the  F 
statistic  from  analysis  of  covariance  exceeded  the  £  statistic  from 
analysis  of  covariance  with  Porter's  adjustment  was  noted  along  with 
the  proportion  of  rejections  by  each  method  of  analysis. 
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A  sign  test  (Siegel,  1956,  p.  63-67)  was  then  performed  for  each 
set  of  conditions  to  test  the  null  hypotheses  in  Chapter  I,  that  is, 
"the  sampling  distributions  of  the  test  statistics  from  analysis  of 
covariance  with  and  without  Porter's  adjustment  will  be  the  same  for 
each  set  of  conditions."  These  tests  were  run  at  the  .01  level  of 
significance. 

The  fraction  of  rejections  noted  (alphas  and  powers)  was  then  used 
as  the  dependent  variable  in  four  factorial  experiments  to  gain  further 
insight  into  what  factors  affected  each  method  of  analysis  being  studied. 
Each  factorial  experiment  had  three  factors.  These  were  reliability 
which  included  six  combinations  of  reliability  used  in  the  study,  sample 
size,  which  included  n  =10  and  n  =  100  subjects  per  group,  and  the 
equality  of  pretest  means,  which  included  a  level  where  both  pretest  means 
v/ere  equal  and  a  level  where  they  differed.  The  layout  of  the  factorial 
experiments  is  illustrated  in  Table  3.  The  factorial  experiment  illus- 
trated is  the  situation  for  which  there  was  no  gain  in  either  group  and 
the  standard  analysis  of  covariance  was  used.  Thus,  the  dependent  variables 
are  the  alpha  values  generated  by  the  computer.  When  significant  main 
effects  or  interactions  occurred,  the  appropriate  a  posteriori  analytical 
procedures  to  locate  the  sources  of  the  variation  were  followed. 
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CHAPTER  IV 
RESULTS 

The.  number  of  times  the  £  statistic  from  the  standard  analysis  of 
covariance  exceeded  the  F  statistic  from  analysis  of  covariance  with 
Porter's  adjustment    is    listed  for  each  set  of  conditions  in  Tables  4, 
5,  6,  and  7.     These  quantities  were  used  as  test  statistics  for  the  sign 
tests  used. to  test  the  hypothesis, "There  is  no  difference  between  the 
sampling  distributions  of  the  test  statistics  from  analysis  of  covariance 
with  and  without  Porter's  adjustment."     This  hypothesis  was  tested  for 
each  of  the  forty-eight  sets  of  conditions  at  the  .01   level  of  signifi- 
cance. 

Siegel   (1956)  stated  that  a  large  sample  test  statistic  for  the 
sign  test  is 

(40)  z  =  X-.5N 

757R~ 

where  N  is  equal  to  the  number  of  pairs  of  observations,  x  is  equal  to 
the  number  of  times  the  first  measurement  of  the  pair  exceeds  the  second 
measurement  of  the  pair,  and  z_  is  the  standard  normal  variate.  For  a 
level  of  .01,  the  null  hypotheses  would  be  rejected  when  z  was  less  than 
-2.33  or  greater  than  2.33.  This  is  equivalent  to  rejecting  the  null 
hypotheses  when  x  was  less  than  949  or  greater  than  1051  and  N  equals 
2000.  Thus,  an  inspection  of  the  last  column  of  Tables  4,  5,  6,  and  7 
reveals  that  the  null  hypothesis  was  rejected  for  each  of  the  forty- 
eight  sets  of  combinations. 

The  analysis  of  variance  summary  tables  avq   presented  in  Tables  8, 
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9,  10,  and  11.     The  analysis  of  variance  summary  table  for  the  case  when 
the  Monte  Carlo  generated  alpha  values  from  the  standard  analysis  of 
covariance  were  used  as  the  criterion  variables  is  presented  in  Table  8. 
The  analysis  of  variance  summary  table  for  the  case  when  the  Monte  Carlo 
generated  alpha  values  from  the  analysis  of  covariance  with  Porter's  ad- 
justment were  used  as  the  criterion  variables  is  presented  in  Table  9. 
The  analysis  of  variance  summary  tables  for  the  cases  when  Monte  Carlo 
generated  powers  were  used  as  the  criterion  variables  for  standard 
analysis  of  covariance  and  analysis  of  covariance  with  Porter's  adjust- 
ment are  presented  in  Tables  10  and  11,  respectively.     Each  analysis  of 
variance  table  includes  analyses  of  the  simple  effects  where  they  are 
warranted.     Scheffe's  S  method  for  testing  linear  contrasts  is  included 
in  Table  8.     These  linear  contrasts  are  defined  in  Table  12.     Each  F 
statistic  which  exceeds  the  critical   value  at  the  .05  level   is  denoted 
by  an  asterisk. 

A  result  of  particular  interest  is  applicable  to  analysis  of  co- 
variance  both  with  and  without  Porter's  adjustment.     That  is,  when 
a  spuriously  high  fraction  of  significant  £  statistics  occurred  when 
there  was  a  mean  gain  of  zero  in  both  groups  and  the  pretest  means 
differed,  the  adjusted  posttest  means  indicated  that  the  gain  was  in 
favor  of  the  group  having  the  largest  pretest  mean.     This  result  was 
more  pronounced  for  lower  reliabilities. 

When  the  gain  group  had  a  positive  gain,  the  no-gain  group  had  a 
mean  gain  of  zero,  and  the  no-gain  group  also  had  a  larger  pretest  mean, 
a  similar  situation  occurred.     In  this  situation,  when  a  significant  £ 
statistic  occurred,  the  adjusted  posttest  means  usually  indicated  the 
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no-gain  group  had  recorded  the  larger  gain.     Again,  these  results  be- 
came more  pronounced  as  the  reliability  of  the  scores  were  reduced. 
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TABLE  8 

ANOVA  SUMMARY  TABLE  USING  MONTE  CARLO  GENERATED  ALPHAS  FROM  THE 
STANDARD  ANALYSIS  OF  COVARIANCE  AS  THE  CRITERION  VARIABLES 


SUM  OF 

DEGREES  OF 

MEAN 

SOURCE 

SQUARES 

FREEDOM 

SQUARE 

F 

Pretest  Mean 

1.59960 

1 

1.59960 

(544.08) 

PM  at  R-, 

.07049 

1 

.07049 

23.98* 

PM  at  R2 

.33582 

1 

.33582 

114.22* 

PM  at  R3 

.40513 

1 

.40513 

137.80* 

PM  at  R4 

.16687 

1 

.16687 

56.76* 

PM  at  Rn 

.33466 

1 

.33466 

113.83* 

PM  at  Rg 

.42510 

1 

.42510 

144.59* 

PM  at  SSio 

.11525 

1 

.11525 

39 . 20* 

PM  at  SS^QQ 

2.10000 

1 

2.10000 

714.29* 

Reliability 

.10996 

5 

.02199 

(7.48) 

R  at  PMi 

.00008 

5 

.00001 

<1.00 

R  at  PM2 

.22278 

5 

.04455 

15.15* 

n 

.00018 

1 

.00018 

<1.00 

4-2  ■ 

.10306 

1 

.10306 

35.05* 

"^'3 

.73933 

1 

.73933 

251.47* 

*4 

.04743 

1 

.04743 

16.13* 

Sample  Size  , 

.59977 

1 

.59977 

(204.00) 

SS  at  PM-, 
SS  at  PM2 

.00005 

1 

.00005 

<1.00 

1.21540 

1 

1.21540 

413.40* 

PM  X  R 

.11291 

5 

.02258' 

7.68* 

PM  X  SS 

.61568 

1 

.61568 

209.41* 

R  X  SS 

.01470 

5 

.00294 

1.00 

Residual 

.01468 

5 

.00294 

Total 


3.06730 


23 


*Sample  statistic  greater  than  critical  value  at  .05  level 
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TABLE  9 

ANOVA  SUMMARY  TABLE  USING  MONTE  CARLO  GENERATED  ALPHAS 

FROM  ANALYSIS  OF  COVARIANCE  WITH  PORTER'S  ADJUSTMENT 

AS  THE  CRITERION  VARIABLES 


SOURCE 

SUM  OF 
SQUARES 

DEGREES  OF 
FREEDOM 

MEAN 
•  SQUARE 

F 

Pretest  Mean 

.45844 

1 

.45844 

(25.08) 

PM  at  SS^o 

.00832 

1 

.00832 

<T.OO 

PM  at  SS-jQQ 

.75050 

1 

.75050 

41.06* 

Reliability 

.17552 

5 

.03510 

1.92 

Sample  Size 

.33018 

1 

.33018 

(18.06) 

SS  at  PM-, 

.00035 

1 

.00035 

<1.00 

SS  at  PM2 

.63021 

1 

.63021 

34.48* 

PM  X  R 

.15727 

5 

.03145 

1.72 

PM  X  SS 

.30038 

1 

.30038 

16.43* 

R  X  SS 

.10189 

5 

.02038 

1.11 

Residual 

.09139 

5 

.01828 

Total 


1.61507 


23 


''Sample  statistic  greater  than  critical  value  at  .05  level 
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TABLE  10 

ANOVA  SUMMARY  TABLE  USING  MONTE  CARLO  GENERATED  POWERS 

FROM  THE  STANDARD  ANALYSIS  OF  COVARIANCE 

AS  THE  CRITERION  VARIABLES 


SOURCE 

SUM  OF 
SQUARES 

DEGREES  OF 
FREEDOM 

MEAN 
SQUARE 

F 

Pretest  Mean 

.97768 

1 

.97768 

(85.54) 

PM  at  SS^o 

.01620 

1 

.01620 

1.42 

PM  at  SS^QQ 

1.61334 

1 

1.61334 

141.15* 

Reliability 

.12431 

5 

.02486 

2.17 

Sample  Size 

.65076 

1 

.65076 

(56.93) 

SS  at  PM^ 

.00001 

1 

.00001 

<1.00 

SS  at  PM2 

1 . 30482 

1 

1 . 30482 

114.16* 

PM  X  R 

.19910 

5 

.03982 

3.48 

PM  X  SS 

.65406 

1 

.65406 

57.22* 

R  X  SS 

.05665 

5 

.01133 

<1.00 

Residual 

.05714 

5 

.01143 

Total 


2.71970 


23 


*Saniple  statistic  greater  than  critical   value  at  .05  level 
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TABLE  11 

ANOVA  SUMMARY  TABLE  USING  MONTE  CARLO  GENERATED  POWERS 

FROM  ANALYSIS  OF  COVARIANCE  WITH  PORTER'S  ADJUSTMENT 

AS  THE  CRITERION  VARIABLES 


SOURCE 

SUM  OF 
SQUARES 

DEGREES  OF     , 
FREEDOM 

MEAN 
SQUARE 

F 

Pretest  Mean 

.15714 

1 

.15714 

(6.48) 

PM  at  SS^Q 

.00157 

1 

.00157 

<1.00 

PMatSS^OQ 

.36018 

1 

.36018 

14.85* 

Reliability 

.14179 

5 

.02836 

1.17 

Sample  Size 

.23285 

1 

.23285 

(9.60) 

SS  at  PH^ 

.00046 

1 

.00046 

<1.00 

SS  at  PM2 

.43701 

1 

.43701 

18.01* 

PM  X  R 

.16990 

5 

.03398 

1.40 

PM  X  SS 

.20461 

1 

.20461 

8.43* 

R  X  SS 

.13367 

5 

.02673 

1.10 

Residual 

.12132 

5 

.02426 

Total 


1.16127 


23 


'^Sample  statistic  greater  than  critical   value  at  .05  level 
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TABLE  12 

COEFFICIENTS  FOR  THE  LINEAR  CONTRASTS  USED  WHEN 
RELIABILITY  WAS  SIGNIFICANT  IN  THE  ANALYSIS  OF  VARIANCE 


LEVELS  OF  RELIABILITY 


RELIABILITY  OF  GAIN  GROUP 
RELIABILITY  OF  NO  GAIN  GROUP 


90 

70 

50 

76 

50 

42 

90 

70 

50 

90 

70 

50 

1 

1 

1 

1 

1 

1 

3 

3 

3 

"3 

"3 

"■3 

1 

1 

0 

1 

1 

0 

2" 

"2" 

Z 

T 

1 

0 

1 

T 

0 

1 

2 

-T 

2 

"2 

0 

1 

1 

0 

1 

1 

2 

"2 

2 

"2 

H-l 


CONTRAST 


CHAPTER  V 
DISCUSSION 

In  general,  the  results  of  this  study  support  the  positions  of 
Lord  (1967,  1969),  Campbell   and  Erlebacher  (1970),  and  O'Connor  (1972) 
:  with  respect  to  their  warnings  about  the  implications  of  analysis  of 
covariance  using  unreliable  test  scores.     The  study  shoves  that  even  when 
there  is  no  gain  in  either  group,  a  much  higher  fraction  of  rejections 
occur  than  would  be  expected.     The  fraction  of  rejections  is  even  more 
extreme  when  the  reliability  is   .70  or  lower  and  the  pretest    means 
differ.     In  addition.  Porter's  adjustment  seems  to  offer  little  improve- 
ment. 

Comparison  of  the  Two  Methods  of  Analysis 

The  rejection  of  all   forty-eight  null   hypotheses  concerning  the 
equality  of  the  sampling  distributions  for  the  two  methods  of  analysis 
shows  that  there  is  a  difference  in  the  results  obtained  from  analysis  '-'^ 
of  covariance  and  analysis  of  covariance  with  Porter's  adjustment.     A 
closer  examination  shows  that  these  differences  are  more  extreme  when 
the  pretest  means  of  the  two  groups  differ  and  the  reliabilities  are  '' 

low.     Further  study  shows  that  although  the  two  methods  of  analysis  are 
different,  neither  method  does  an  adequate  job  of  modeling  reality,  that 
is,  both  methods  tend  to  produce  erroneous  proportions  of  type  I  and  II 
errors  v/hen  pretest  means  differ  and  the  reliabilities  are  low.     When  the 
pretest  means  are  equal,  the  power  of  the  tests  seem  to  be  directly  re- 
lated in  a  positive  manner  to  the  reliabilities  when  other  variables  are 
held  constant. 
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Possibly  the  most  far  reaching  results  were  a  function  of  re Ij Ability-. 
The  data  indicated  that  incorrect  decisions  about  which  group  had  the 
larger    gain  could  be  made  using  either  method  of  analysis  when  the  pre- 
test means  of  the  two  groups  differed  and  the  reliabilities  were  low. 
When  the  pretest  mean  of  the  gain  group  was|  less  than  that  of  the  no-gain 
group,  the  adjusted  posttest  means  stiowed  that  the  no-gain  group  was 
superior  both  when  the  mean  gain  was  zero  in  both  groups  and  when  the 
mean  gain  was  positive  only  in  the  gain  group.     This  possibility  was 
pointed  out  by  Lord  (1967)  land  Campbell   and  Erlebacher  (1970). 

■■■■-•  I  ■■'.-' 

Factors  that  Affect  Alpha  and  Beta 

The  factorial  experiments  using  the  computer  generated  alphas  and 
betas  allow  one  to  infer  which  factors  affect  the  levels  of  type  I  and 
type  II  errors.     Using  Monte  Carlo  generated  alphas  from  the  standard 
analysis  of  covariance  in  a  factorial   experiment  indicated  that  an  inter- 
action between  pretest  means  and  reliability  and  an  interaction  between 
sample  size  and  pretest  means  affected  the  alphas  significantly.     Further 
analyses   (simple  effects)  showed  that  the  pretest  mean  factor  was  signif- 
icant at  every  level  of  reliability.     Both  reliability  and  sample  size 
were  significant  when  the  pretest  means  differed.     The  tests  of  linear 
contrasts  showed  that  there  was  no  significant  difference  between  equal 
reliabilities  and  unequal   reliabilities  when  the  pretest  means  differed, 
but  there  were  significant  differences  among  the  levels  of  reliability 
when  the  pretest  means  differed.     These  results  indicate  that  a  difference 
in  reliabilities  between  groups,  thus  a  difference  in  slopes,  has  no 
effect,  however,  the  level   of  reliability  does. 

The  other  three  factorial  experiments  indicated  that  interactions 


43 


between  pretest  means  and  sample  size  were  the  major  contributors  to  the 
differing  levels  of  alpha  and  power.  In  all  three  experiments  (See  Tables 
10,  11,  and  12)  the  sample  size  was  significant  when  the  pretest  means 
differed.  Reliability  seems  to  have  a  somewhat  moderate  effect  in  these 
cases. 

Predicting  Alpha  and  Power 

Using  the  results  of  this  study,  a  regression  equation  can  be  set 
up  to  predict  the  experimental  probability  of  a  type  I  error  or  the 
experimental  probability  of  rejecting  a  false  null  hypotheses  when  the 
established  probabilities  are  .05  and  .50  respectively.  The  basic 
regression  equation  is 

(41)      Y  =  eo+3TRG+32RN+B3S+34M+B5RQRf^+36RGS 
+e7RQM+B3RfjS+3gR[^M+6iQSM+G 

where  • 

Y  =  the  predicted  alpha  or  power, 

Rq  =  the  reliability  of  the  gain  group  scores, 

R^   =  the  reliability  of  the  no  gain  group  scores, 

S  =  sample  size 

M  =  1  if  the  pretest  means  are  equal, 
0  otherwise,  and 

3i  =  the  regression  coefficient,  i. 

Tables  14  and  15  provide  confidence  intervals  for  the  expected 

values  of  alpha  and  power  respectively  for  each  set  of  conditions  noted. 
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TABLE  13 

NINETY-FIVE  PERCENT  CONFIDENCE  INTERVALS  FOR  THE  EXPECTED 
VALUES  OF  ALPHAS  UNDER  SPECIFIED  CONDITIONS 


LOWER 

UPPER 

RELIABILITY 

RELIABILITY 

GROUP 

EQUALITY 

CONFIDENCE 

CONFIDENCE 

OF  GAIN 

OF  NO  GAIN 

SAMPLE 

OF  PRETEST 

LIMIT  OF 

LIMIT  OF 

GROUP  SCORES 

GROUP  SCORES 

SIZE 

MEANS* 

ALPHA 

ALPHA 

.90 

.90 

10 

1 

-  .000 

.137 

.90 

.90 

10 

0 

.000 

.137 

.90 

.90 

100 

1 

.000 

.100 

.90 

.90 

100 

0 

.556 

.741 

.70 

.70 

10 

1 

.017 

.162 

.70 

.70 

10 

0 

.  1 08 

.325 

.70 

.70 

100 

1 

.000 

.141 

.70 

.70 

100 

0   . 

.780 

.945 

.50 

.50 

10 

1 

.000 

.125 

.50 

.50 

10 

0 

.297 

.451 

.50 

.50 

100 

1 

.000 

.120 

.50 

■  .50 

100 

0 

.932 

1.000 

.76 

.90 

10 

1 

.000 

.136 

.76 

.90     . 

10 

0 

.043 

.223 

.76 

.90 

100 

1 

.036 

.143 

.76 

.90 

100 

0 

.  .691 

.871 

.60 

.70 

10 

1 

.003 

.136 

.60 

.70 

10 

0 

.228 

.361 

.60 

.70 

100 

1 

.014 

.146 

.60 

.70 

100 

0 

.879 

1.000 

.42 

.50 

10 

1 

.000 

.097 

.42 

.50 

10 

0 

.307 

.472 

.42 

.50 

100 

1 

.000 

.117 

.42 

.50 

100 

0 

.967 

1.000 

*If  1,  the  pretest  means  are  equal  and  if  0,  the  pretest  means 
are  not  equal . 
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TABLE  14 

NINETY-FIVE  PERCENT  CONFIDENCE  INTERVALS  FOR  THE  EXPECTED 
VALUES  OF  POWER  UNDER  SPECIFIED  CONDITIONS 


LOWER 

UPPER 

RELIABILITY 

RELIABILITY 

GROUP 

EQUALITY 

CONFIDENCE 

CONFIDENCE 

OF  GAIN 

OF  NO  GAIN 

SAMPLE 

OF 

PRETEST 

LIMIT  OF 

LIMIT  OF 

GROUP  SCORES 

GROUP  SCORES 

SIZE 

MEANS* 

POWER 

POWER 

.90 

.90 

10 

.000 

.300 

.90 

.90 

10 

0 

.000 

.115 

.90 

.90 

100 

.000 

.191 

.90 

.90 

100 

0 

.351 

.666 

.70 

.70 

10 

.019 

.266 

.70 

.70 

10 

0 

.048 

.295 

.70 

.70 

100 

.000 

.236 

.70 

.70 

100 

0 

.677 

.924 

.50 

.50 

10 

.000 

.176 

.50 

.50 

10 

0 

.155 

.417 

.50 

.50 

100 

.000 

.  224 

.50 

.50 

100 

0 

.863 

1.000 

.76 

.90 

10 

.000 

.265 

.76 

.90 

'   10 

0 

.000 

.199 

.76 

.90 

100 

.000 

.235 

.76 

.90 

100 

0 

.523 

.829 

.60 

.70 

10 

.000 

.210 

.60 

.70 

10 

0 

.097 

.323 

.60 

.70 

100 

.010 

.236 

.60 

.70 

100 

.782 

1 .000 

.42 

.50 

10 

.000 

.130 

.42 

.50 

10 

.156 

.438 

.42 

.50 

100 

.000 

1.000 

.42 

.50 

100 

0 

.909 

*If  1,  the  pretest  means  are  equal  and  if  0,  the  pretest  means 
are  not  equal . 
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The  predicted  value  of  alpha  or  power  can  be  found  by  using  the 
data  found  in  Tables  4,  5,  6,  and  7  to  fit  regression  equation  (41)  to 
obtain  the  following  estimated  regression  parameters: 

(42)  alpha  =  .228  +  .395Rq  +  .332R,^  +  .008  S  -  .661M 

-1.09  RqR,^  -  .004  RqS  +  .619  RgM 
-.003  R|^jS  +  .195  R|^M  -  .007  SM 
and 

(43)  power  =  .045  +  .536Rg  +  .404  R|^  +  .010  S  -  .700  M 

-1.228  RqR,^  -  .006  RqS  +  .846  RgM 
+  .002  RfjS  +  .218  RfjM  -  .007  SM. 
A  confidence  interval  for  the  predicted  values  of  alpha  and  power 
could  be  obtained  in  the  usual  manner. 

A  Direction  for  Further  Research 

The  results  of  this  study  combined  with  other  research  (Lord,  1967; 
Campbell  and  O'Connor,  1972),  indicates  a  need  for  further  study  and  the 
development  of  a  robust  method  for  comparing  groups  of  differing  ability 
when  the  scores  are  not  perfectly  reliable.  Any  new  technique  which  is 
proposed  should  first  be  investigated  under  known  conditions,  either 
analytically,  or  by  Monte  Carlo  techniques.  This  would  insure  that 
another  inappropriate  method  is  not  used  for  the  evaluation  of  compensa- 
tory education  projects. 


CHAPTER  VI 
SUMMARY 

This  study  was  designed  to  determine  if  either  analysis  of  covar-     ^. 
iance  or  analysis  of  covariance  with  Porter's  adjustment  is  an  appropri- 
ate analytical   procedure  for  evaluating  educational   pretest-posttest 
experiments.     In  particular,  these  methods:  were  compared  with  respect  to 
their  use  in  the  analysis  of  compensatory  education  projects  where  the 
groups  may  differ  in  ability. 

The  study  was  carried  out  by  computer  generating  2000  sets  of 
normal   data  under  forty-eight  sets  of  predetermined  conditions  of  reli- 
ability, sample  size,  gain,  and  equality  of  pretest  means.     Each  of  the 
2000  sets  of  data  for  each  set  of  conditions  was  then  analyzed  using  both 
standard  analysis  of  covariance  and  analysis  of  covariance  with  Porter's 
adjustment.     A  sign  test  was  used  to  compare  the  two  methods  of  analysis 
under  each  of  the  forty-eight  sets  of  conditions.     It  was  concluded  that    -^ 
the  two  methods  of  analysis  yielded  different  results. 

Factors  affecting  the  two  methods  of  analysis  were  then  studied 
separately  using  the  computer  generated  alphas  and  powers  as  criterion 
variables  in  four  factorial   experiments.     The  factors  included  reliability 
at  six  levels,  sample  size  at  two  levels,  and  the  equality  of  pretest 
means  at  two  levels.     From  these  experiments,  it  was  concluded  that  pre-    "^ 
test  means  interacting  with  sample  size  and  sometimes  with  reliability 
were  significant  factors.     More  specifically,  sample  size  was  statistically 
significant  in  each  case  where  the  pretest  means  differed.     Also,  pretest 
means  were  significant  at  ^M^r^  level  of  reliability  for  the  computer 
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generated  alphas  produced  by  the  standard  analysis  of  covariance. 
It  was  also  learned  that  when  the  pretest  means  differ,  both 
standard  analysis  of  covariance  and  analysis  of  covariance  with  Portier's 
adjustment  produced  erroneous  results  with  respect  to  which  group  if 
either  had  a  gain.  When  both  groups  had  a  mean  gain  of  zero  and  the 
pretest  means  differed,  significant  results  usually  indicated  that  the 
group  with  the  larger  pretest  mean  had  the  gain.  This  would  correspond 
to  the  control  group  sampled  from  the  general  population  being  credited 
with  the  gain  in  a  compensatory  education  experiment.  When  there  was  a 
gain  in  only  one  group  and  the  pretest  mean  was  lower  in  that  group,  the 
analyses  still  indicated  that  the  other  group  had  the  gain. 

The  results  of  this  study  point  to  the  recommendations  that  analysis 
of  covariance  with  or  without  Porter's  adjustment  should  be  approached    ^ 
with  caution  when  the  reliabilities  are  below  .90  and  the  pretest  means 
(covariate  means)  are  likely  to  be  different  for  the  groups. 
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FCRTRAN  PRCGRAN  WHICH  PERFORMED  DATA  GFNERATIOiN  AND  ANALYSIS 


DIh'ENSICN  EIGN(IOO),  E2GN(  100)  ,GAINGN{  ICO)  tXX(  2G0  )  ,  YY  (  200  )  , 
»X(100)  .ElNGdOO),  E2NG( 100 ),GAINNC( I0a),TG(100) ,TN(I00) 
REAL  .NS[3,NSE,MSBP,MSeP 


C 

c 
c 
c 
c 
c 
c 
c 
c 
c 

99 
100 
101 


C 

c 
c 

102 


103 

104 

105 

C 
C 
C 


.  NGRCUP=0 

INPUT  CF  GRCUP  PARAMETERS  WHEREO 

RELGN  REPRESENTS  THE  RELIABILITY  OF  THE  GAIN  GROUP  DATA 
RELNG  REPRESENTS  T HE  R EL  I AB  IL  I T Y  OF  THE  NO  GAIN  GROUP 
NPERG  REPRESENTS  THE  NUMBER  OF  OBSERVATIONS  PER  GROUP 
GBAR  REPRESENTS  THE  MEAN  GAIN  FOR  GAIN  GROUP 
PR^NG  REPRESENTS  THE  GAIN  GROUP  PRETEST  MEAN 
FR^NNG  REPRESENTS  THE  NO  GAIN  GROUP  PRETEST  MEAN 
ISEEC  REPRESENTS  THE  SEED  FOR  RANDOM  NUMBER  GENERATOR 

READ  (5,99)  I S EED , NGP , NS PL 

FORMAT  (  IH,2X,2I5) 

READ  15,101)  RELGN, RELNG, NPERG, GDAR,PRMNG,PRMNNG 

FORMAT  {2F3.2, 13, F3.2,2F3.0) 

NGRCUP  =  NGl-iGUP  +  l 

NSAh'F  =  0 

NF10=0 

NF 100=0 

NFP10=0 

NFP100=0 

HEADER  CARD  FOR  NEW  SET  OF  PARAMETERS 

WRITE  (6,102) 

FORMAT  (• l',T43, ' F  STATISTICS  BASED  ON  THE  FOLLOWING  PARME' 
•  ,«TERSM 

WRITE  (6,103) 

FORMAT  ( •0',T22, 'RELGN  •,2X, 'RELNG  •,2X, 'NPERG  ',2X,'  GBAR' 
»,'  'i^X.'PRMNG  •  ,2X,  'PRMNNG'  ) 

WRITE  (6,104)  RELGN, RELNG, NPERG, GBAR, PRMNGfPRMNNG 

FORMAT  (1X,T22,2F8.5, I5,3X,3F8.3) 

WRITE  (6,105) 

FORMAT  ( '0' ,T45, "SAMPLE  NUMB ER' , 5X ,' STANDARD  F ', 5X ,' PORTER ' 
», 'S  F') 

COMPUTE  VARIANCE  COMPONENTS 

VARE1G=100»{1-RELGN) 

VARE1N=100«(1-RELNG) 

VARTG=100«RELGN 

VARTK=100"RELNG 

VARGNG=.04»VAf?TG 

VARGNN=.04«VARTN 
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C 
106 

C 
C 
C 
C 

c 
c 


300 

c 
c 
c 


301 
C 

c 
c 


302 
C 
C 
C 


303 
C 
C 
C 


304 

C 

C 

C 

C 

C 

C 


VARE2G=(V/iRTG+VARGNG)/RELGN-VARTG-yARGNG 
VARE2N=( VARTN+VARGNN ) /RELNG-V ARTN-VARGNN 

CCNTINUc 
NSAMP=;NSA^P  +  1 

GENERATICN  CF  CATA  FOR  GAIN  GROUP 


TRUE  SCORES 

CALL    RAKGENCNPERG,  ISEECX) 

DO    3C0    I=1,NPERG 

TG(I )=X( I ) •SQRTIVARTGI+PRMNG 

CCNTINUE 

PRETEST  ERROR  SCORES 

CALL  RANGEN(NPERG,ISEED,X) 
CO  301  I=1,NPERG 
eiGN(I)=X(I)»SeRT(VARElG) 
CCNTINUE 

POSTTEST  ERROR  SCORES 

CALL  RANGEN(NPERG,.IS'EEn,,X) 

DC  302  I^l.NPERG 

E2GN( I)=XI I)»SCRT(VARE2G) 

CONTINUE 

GAIN  SCORES 

CALL    RANGEMNPERG,  ISEEO.X  ) 

CO    303  ■  I  =  1,NPERG 

GAINGNi I )=X( I )«SQRT{ VARGNG)+GBAR 

CONTINUE 

PRETEST  ANC  POSTTEST  SCORES 

[JO  30 A  I  =  1,NPERG 

XXd  )=TG{  I)+E1GN(  I  ) 

YY( I  l=TG{  I)+GAINGN(  I  )  +  E2GN( I  ) 

CCNTINUE 


GENERATION  OF  CATA  FOR  NO  GAIN  GROUP 

TRUE  SCORES 

CALL  RANGEN(NPERG,ISEED,X) 
00  400  I=1,NPERG 
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400 
r. 
c 
c 


401 

c 
c 
c 


402 

C 

C 

c 


403 
C 
C 
C 


404 

C 

C 

C 

C 

C 
C 
C 


TN(  I  )nX(  I  )«SQRT(VARTN)+PRMNNG 

CCMINUC 

PRETEST  ERRCR  SCORES 

CALL  PANGENCMPERG,  ISEECX) 
DC  .401  I  =  1,NPERG 
.  E1NG(  I)  =  Xn  )»SGRTIVAPE1N) 
CCNTINUE 

POSTTEST  ERROR  SCORES 

CALL  PANGEN(NPERG,  ISEED.X  ) 

CC  402  I=1,NPERG 

E2NG( I)=X(I )»SCRT(VARE2N) 

CCNTINUt 

GAIN  SCORES 

CALL  RANGEN(NPERG, ISEED.X) 

DC  403  1=1,NPERG 

GAINNG( I  )=X( I  )«SQRT{ VARGNN) 

CCNTlKUt: 

PRETEST  ANC  POSTTEST  SCORES 

DO  404  I=1,NPERG 

L=NPERG+I 

XX(L)=TN( I)+E1NG(  I  ) 

YY{L)=TN( I )+GAINNG{  I  )  +  E2NG(  I  ) 

CCNTINUE 


STANCARC  ANALYSIS  OF  COVARIANCE  COMPUTATIONS 

N=2*NPERG 

IMTIALIZATICN 

SUNXG=0.0 

SUNX2G=0.C 

SU^YG=0.0 

SUMY2G=0.0 

SUNXYG=O.C 

SUNXN=a.O 

50^X2^=0.0 

SUN'YN=0.0 

SUyY2N=0.C 

SUNXYN=0.C 

GROUP  SUMS  AND  SUMS  OF  SQUARES 
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600 
C 
C 
C 


DC    6G0     I=1,NPERG 

sur'XG=sur'XG+xx{i ) 

SUNX2G  =  SUNX2GtXX(  I  )»XX(  I  ) 
.     SUr'YG  =  SUI^YG+YY(  I  ) 

SUNY2G  =  SUNY2G  +  YY(  I  )»YY(  I  ) 
.    SU^'XYG  =  SU^XYG  +  XX(  I  )»YY(  I  ) 

K=NPtRG+I 

SUr'XN^SUNXN+XXCK  ) 

SU^X2N  =  SUr'X2N  +  XX(K)»XX(K) 

SUNYN  =  SUNYN  +  YY (K  ) 

SU^Y2^  =  SU^■Y2^J+YY(K)»YY{K) 

SUr'XYN=SUNXYrj+XX  (K)»YY(K  ) 

GCNTINUC 

TOTAL  SUMS  AND  SUMS  OF  SQUARES 

TSUVX=SUNXG+SUMXN 

TSUMX2=SU^X2G+SUMX2N 
TSUI^Y  =  SU^'YG  +  SUMY^I 
TSUNY2=SUNY2G+SUMY2N 
TSU^XY=SUNXYG+SUMXYN 

COMPUTE  TOTAL  SUMS  OF  SQUARES 

CFX=TSUMX«TSU^X/N 

CFY=TSUMY«TSU^Y/N 

CFXY  =  TSUMX«TSU^'Y/N 

TXX=TSUMX2-CFX 

TYY=TSUKY2-CFY 

TXY=TSU^XY-CFXY 

COMPUTE  BETWEEN  GROUPS  SUMS  OF  SQUARES 

BXX=  (SUMX'G*SUMXG4-SUMXN»SUMXN  )/NPERG-CFX 
BYY=(SUMYG«SUMYG+SUMYN*SUMYN)/NPERG-CFY 
BXY= (SUMXG»SUMYG+SUMXN»SUMYN )/NPERG-CFXY 

COMPUTE  ERROR  SUMS  OF  SQUARES 

Exx=Txx-exx 

EYY=TYY-eYY 
EXY  =  TXY-e.XY 

COMPUTE  ACJUSTED  SUMS  OF  SQUARES 

TYYACJ=TYY-TXY«TXY/TXX 
£YYACJ=EYY-EXY«EXY/EXX 
BYYACJ=LiYY-{TXY»TXY)/TXX  +  (EXY«EXY)/EXX 
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C 
G 

80C 
C 

C 

801 

802 


CCNPUTE  ACJUSTEC  MEAN  SQUARES 

^Se=BYYACJ/l.C 
ySE  =  EYYA[:j/  (M-3) 

CC^'PUTE  F  STATISTIC 

.  F  =  N'SB/MSE 

ANALYSIS  CF  COVARIANCE  WITH  PORTER'S  ADJUSTMENT 

CCPPUTE  CCRRELATION  BETWEEN  X  AND  Y 

CENCN=TXX«TYY 
RXY=TXY/SGRT( CENOy) 

COMPUTE  SUNS  OF  SQUARES  WITH  PORTER'S  ADJUSTMENTS 

EPCRT=EYYACJ 

TPCRT=TYY-l {RXY»EXY+eXY)»«2)/( (RXY*RXY»EXX)+BXX) 

BPCRT=TPCRT-EPCRT 

COMPUTE  MEAN  SQUARES 

NSEP=EPCRT/I.O 
MSEP=EPCRT/{N-3) 

COMPUTE  F  STATISTIC  WITH  PORTER'S  ADJUSTMENT 

FPORT-MSBP/MSEP 

IF  (F.GT.A.^5)  NFlO=NFin+l 
IF  (F.GT.3.89J  NF  I00  =  KF  1G0+,  1 
IF  (FFORT.GT.^.45)  N FP  10  =  NFP  lC  +  1 
IF  (FFCRT.GT.3.89)  N FP IOO=NFP 1C0+ 1 
WRITE  CUT  RESULTS 

WRITE  16,800)  NSAMP, F,FPORT 

FORMAT  (IX,T50,  I4,T6A, F8.3,T77,F8.3) 

IF  (NSAMP. LT. NSPL  )  GC  TO  106 

WRITE  (6,801) 

FORMAT  (  'C  ,T10,  •NF10.«,.T20,  'NF10C«,T30,  'NFPIO'  ,T40,  'NFPIOO' 
•) 
WRITE  (6,802)  NF 10 , N F 100 , NFP IC ,NFP 100 
FORMAT  (  1X,T10,4(  H,6X  )  ) 
IF  (KGRCUP.LT.NGP)  GC  TO  100 
STOP 
END 
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SUDRCL'TINE  RANGEN  (  K,  I  R  ,.X  ) 

CINENSICN  X(M) 

1  =  1 

CALL  RANCU( IRtJR.Rl) 

1R=  JR 

CALL  PANCU( IR, JR,R2) 

1R  =  JR 

Rl  =  2.G»<Rl-.5  ) 

R2  =  2.0«'{R2-.5  ) 

S2  =  R1  «R1  +  R2«R2 

IF{S2.GT.1.0)  GO  TO  1 

Y  =  SQRT(-2.0»( AL0G(S2)/S2)  ) 

X(I)=(a«Y 

IF(I.EG.N)  GO  to  2 

X(Itl)=R2«Y 

IF(I+1.EC.M)  GO  TO  2 

1  =  1+2 

GC  TC  1 

RETURN 

END 


SUBRCUTINE  RANCU  I  IX  ,  lY  ,.Y  FL  ) 

I Y= IX •65539 

IF(IY)5,6,6 

5  IY=IY+2l^7^836A7+l 

6  YFL=IY 

YFL  =  Y(FL».^656613E-9 

RETURN 

END 
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